Endscope apparatus and program

ABSTRACT

An endoscope apparatus includes an image acquisition section that acquires images in time series, a coefficient calculation section that calculates a correction coefficient for correcting blurring in a depth direction that is a direction along an optical axis of an imaging section, and a depth-direction blurring correction section that performs a depth-direction blurring correction process that corrects blurring in the depth direction on the images acquired in time series based on the correction coefficient.

Japanese Patent Application No. 2010-252555 filed on Nov. 11, 2010, ishereby incorporated by reference in its entirety.

BACKGROUND

The present invention relates to an endoscope apparatus, a program, andthe like.

In recent years, differential diagnosis has been performed using anendoscope apparatus by observing the object in a magnified state. Anoptical zoom process that does not cause a deterioration in the objectimage is generally used for magnifying observation. An electronic zoomprocess performed on image data acquired by a CCD or the like canfurther magnify the object image subjected to the optical zoom process.

For example, JP-A-5-49599 discloses a method that performs a blurringcorrection process by detecting the motion of the end of the scope whenusing an endoscope apparatus that implements magnifying observation.According to this method, a moving amount detection means that detectsthe direction and the angular velocity is provided in the curved sectionof the endoscope, and the blurring correction process is performed basedon the moving direction and the moving distance.

JP-A-2009-71380 discloses a method that detects the motion amount of theobject, and stops the moving image at an appropriate timing by detectinga freeze instruction signal to acquire a still image. According to thismethod, an optimum frozen image with a small amount of blurring isgenerated by detecting the periodicity of the image.

SUMMARY

According to one aspect of the invention, there is provided an endoscopeapparatus comprising:

an image acquisition section that acquires images in time series;

a coefficient calculation section that calculates a correctioncoefficient for correcting blurring in a depth direction that is adirection along an optical axis of an imaging section; and

a depth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction.

According to another aspect of the invention, there is provided aninformation storage medium storing a program that causes a computer tofunction as:

an image acquisition section that acquires images in time series;

a coefficient calculation section that calculates a correctioncoefficient for correcting blurring in a depth direction that is adirection along an optical axis of an imaging section; and

a depth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a configuration example of an endoscope apparatus.

FIG. 2 shows a specific configuration example of a rotary color filter.

FIG. 3 shows the spectral characteristics of a color filter.

FIG. 4 shows a specific configuration example of an image processingsection.

FIG. 5 shows a specific configuration example of a blurring correctionsection.

FIGS. 6A to 6G are views illustrative of a blurring correction process.

FIG. 7 shows an example of feature point data used for a blurringcorrection process.

FIG. 8 shows an example of a flowchart of an image processing program.

FIG. 9 shows a specific example of a flowchart of a blurring correctionprocess.

FIG. 10 shows a second configuration example of an endoscope apparatus.

FIG. 11 shows a second specific configuration example of an imageprocessing section.

FIG. 12 shows a second specific configuration example of a blurringcorrection section.

FIG. 13 shows a second specific example of a flowchart of a blurringcorrection process.

FIG. 14 shows a third configuration example of an endoscope apparatus.

FIG. 15 shows a third specific configuration example of an imageprocessing section.

FIG. 16 shows a third specific configuration example of a blurringcorrection section.

FIG. 17 is a system configuration diagram showing the configuration of acomputer system.

FIG. 18 is a block diagram showing the configuration of a main bodyincluded in a computer system.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

When implementing magnifying observation, the effects of blurringincrease as the magnification increases. For example, when using anendoscope apparatus that observes an object inside the body cavity, theinternal organs such as the gullet make a motion due to the heartbeat.Therefore, blurring of the object occurs in the depth direction, so thatthe observation capability (e.g., visibility) deteriorates.

Several aspects of the invention may provide an endoscope apparatus, aprogram, and the like that can suppress blurring of a moving image inthe depth direction.

According to one embodiment of the invention, there is provided anendoscope apparatus comprising:

an image acquisition section that acquires images in time series;

a coefficient calculation section that calculates a correctioncoefficient for correcting blurring in a depth direction that is adirection along an optical axis of an imaging section; and

a depth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction.

According to one aspect of the invention, images are acquired in timeseries, the correction coefficient for correcting blurring in the depthdirection is calculated, and the process that corrects blurring in thedepth direction is performed on the images acquired in time series basedon the correction coefficient. This makes it possible to suppressblurring of a moving image in the depth direction.

Exemplary embodiments of the invention are described below. Note thatthe following exemplary embodiments do not in any way limit the scope ofthe invention laid out in the claims. Note also that all of the elementsof the following exemplary embodiments should not necessarily be takenas essential elements of the invention.

1. Method

An outline of a depth-direction blur correction process according to oneembodiment of the invention is described below. When the doctor hasinserted an endoscope end into the digestive tract, the doctor searches(screens) a lesion by moving the end of the endoscope. When the doctorhas found a lesion area, the doctor magnifies the object using anoptical zoom process or an electronic zoom process, and observes thelesion area without moving the end of the endoscope.

In this case, blurring of the object image may occur due to the motionof the object and the motion of the imaging section. As shown in FIG. 1,such blurring includes blurring in the optical axis direction of animaging section 200, and blurring in the optical axis direction isobserved as blurring in the depth direction of the image. Blurring inthe depth direction occurs due to the motion of the internal organscaused by the heartbeat and the peristaltic motion of the digestivetract, for example. Since the amount of blurring of the object imageincreases depending on the magnification employed during magnifyingobservation, the amount of blurring of a moving image increases as themagnification increases.

In one embodiment of the invention, the electronic zoom or optical zoommagnification is adjusted corresponding to a change in magnification ofimages acquired in time series (see FIGS. 6A and 6B) so that the size ofthe object within the images acquired in time series is constant (seeFIG. 6C). This makes it possible to reduce the amount of blurring of themoving image in the depth direction due to the motion of the internalorgans, so that the visibility of the moving image can be improved.

2. First Configuration Example of Endoscope Apparatus

FIG. 1 shows a first configuration example of an endoscope apparatusthat corrects blurring in the depth direction. The endoscope apparatus(endoscope system) includes a light source section 100, an imagingsection 200, a control device 300 (processor section in a narrow sense),a display section 400, and an external I/F section 500.

The light source section 100 emits illumination light that illuminatesan object. The light source section 100 includes a white light source101, a light source aperture 102, and a light source aperture driversection 103 that drives the light source aperture 102. The light sourcesection 100 also includes a rotary color filter 104 that has a pluralityof spectral transmittances, a rotation driver section 105 that drivesthe rotary color filter 104, and a condenser lens 106 that focuses lighthaving spectral characteristics due to the rotary color filter 104 on anincident end face of a light guide fiber 201.

The light source aperture driver section 103 adjusts the intensity oflight by opening or closing the light source aperture 102 based on acontrol signal output from a control section 320 included in the controldevice 300.

As shown in FIG. 2, the rotary color filter 104 includes a red colorfilter 601, a green color filter 602, a blue color filter 603, and arotary motor 803. The color filters 601 to 603 have spectralcharacteristics shown in FIG. 3, for example.

The rotation driver section 105 rotates the rotary color filter 104 at agiven rotational speed in synchronization with an imaging period of animaging element 206 based on a control signal output from the controlsection 302 included in the control device 300. For example, whenrotating the rotary color filter 104 at 20 rotations per second, thecolor filters 601 to 603 cross incident white light every 1/60th of asecond. Therefore, the imaging element 206 completes acquisition andtransfer of an image of reflected light in each color (R, G, or B) every1/60th of a second. The imaging element 206 is a monochrome imagingelement, for example. In this case, an R image, a G image, and a B imageare frame-sequentially acquired (captured) every 1/60th of a second.

The imaging section 200 is formed to be elongated and flexible (i.e.,can be curved) so that the imaging section 200 can be inserted into abody cavity or the like. The imaging section 200 includes the lightguide fiber 201 that guides light focused by the light source section100, and an illumination lens 202 that diffuses light guided by thelight guide fiber 201, and illuminates the object. The imaging section200 also includes an objective lens 203 that focuses reflected lightfrom the object, a variable aperture 204, and an objective aperturedriver section 205 that opens or closes the variable aperture 204 undercontrol of the control section 320. The imaging section 200 alsoincludes an imaging element 206 for detecting the focused reflectedlight, and an A/D conversion section 207 that converts an analog signalobtained by a photoelectric conversion process performed by the imagingelement 206 into a digital signal. The imaging element 206 is a singlemonochrome imaging element, for example. The imaging element 206 may beimplemented by a CCD or a CMOS sensor.

The control device 300 controls each element of the endoscope apparatus,and performs image processing. The control device 300 includes an imageprocessing section 310 and the control section 320.

An image (image signal) converted into digital data by the A/Dconversion section 207 is transmitted to the image processing section310. The image processed by the image processing section 310 istransmitted to the display section 400.

The control section 320 is connected to the light source aperture driversection 103, the rotation driver section 105, the objective aperturedriver section 205, the imaging element 206, the image processingsection 310, and the external I/F section 500, and controls the lightsource aperture driver section 103, the rotation driver section 105, theobjective aperture driver section 205, the imaging element 206, theimage processing section 310, and the external I/F section 500.

The display section 400 displays an image or a moving image input fromthe image processing section 310. The display section 400 includes adisplay device (e.g., CRT or liquid crystal monitor) that can display amoving image.

The external I/F section 500 is an interface that allows the user toinput information to the endoscope apparatus (imaging apparatus), forexample. The external I/F section 500 includes a power supply switch(power supply ON/OFF switch), a shutter button (imaging (photographing)operation start button), a mode (e.g., imaging (photographing) mode)switch button, and the like. The external I/F section 500 transmits theinput information to the control section 320.

3. Image Processing Section

The details of the image processing section 310 are described below.FIG. 4 shows a specific configuration example of the image processingsection 310. The image processing section 310 includes a preprocessingsection 311, a demosaicing section 312, a blurring correction section313, and a post-processing section 314.

The A/D conversion section 207 is connected to the preprocessing section311. The preprocessing section 311 is connected to the demosaicingsection 312. The demosaicing section 312 is connected to the blurringcorrection section 313. The blurring correction section 313 is connectedto the post-processing section 314. The post-processing section 314 isconnected to the display section 400. The control section 320 isconnected to the preprocessing section 311, the demosaicing section 312,the blurring correction section 313, and the post-processing section314, and controls the preprocessing section 311, the demosaicing section312, the blurring correction section 313, and the post-processingsection 314.

The preprocessing section 311 performs an OB clamp process, a gaincontrol process, and a WB correction process on the digital image inputfrom the A/D conversion section 207 using an OB clamp value, a gaincorrection value, and a WB coefficient stored in the control section320. The preprocessed image is transmitted to the demosaicing section312.

The demosaicing section 312 performs a demosaicing process on theframe-sequential R, G, and B images processed by the preprocessingsection 311 based on a control signal input from the control section320. The demosaiced images are transmitted to the blurring correctionsection 313.

The blurring correction section 313 performs a blurring correctionprocess on the demosaiced time-series images. The blurring correctionsection 313 corrects blurring in the depth direction and blurring in theplanar direction as the blurring correction process. The image subjectedto the blurring correction process is transmitted to the post-processingsection 314.

The post-processing section 314 performs a grayscale conversion process,a color process, a contour enhancement process, and an enlargementprocess using a grayscale conversion coefficient, a color conversioncoefficient, a contour enhancement coefficient, and an enlargementfactor stored in the control section 320. The post-processed image istransmitted to the display section 400.

4. Blurring Correction Section

The details of the blurring correction section 313 are described below.FIG. 5 shows a specific configuration example of the blurring correctionsection 313. The blurring correction section 313 includes a firststorage section 701, a correction start detection section 702, acoefficient calculation section 703, a depth-direction blurringcorrection section 704, a trimming section 705, a second storage section706, a determination section 707, and a planar-direction blurringcorrection section 708.

The demosaicing section 312 is connected to the first storage section701 and the correction start detection section 702. The first storagesection 701 is connected to the correction start detection section 702and the coefficient calculation section 703. The correction startdetection section 702 is connected to the coefficient calculationsection 703 and the post-processing section 314. The coefficientcalculation section 703 is connected to the depth-direction blurringcorrection section 704. The depth-direction blurring correction section704 is connected to the trimming section 705. The trimming section 705is connected to the second storage section 706 and the determinationsection 707. The second storage section 706 is connected to thedetermination section 707. The determination section 707 is connected tothe planar-direction blurring correction section 708 and thepost-processing section 314. The planar-direction blurring correctionsection 708 is connected to the post-processing section 314. The controlsection 320 is connected to the correction start detection section 702,the coefficient calculation section 703, the depth-direction blurringcorrection section 704, the trimming section 705, the determinationsection 707, and the planar-direction blurring correction section 708,and controls the correction start detection section 702, the coefficientcalculation section 703, the depth-direction blurring correction section704, the trimming section 705, the determination section 707, and theplanar-direction blurring correction section 708.

The first storage section 701 stores the image (image signal) input fromthe demosaicing section 312.

The correction start detection section 702 determines whether or not tostart the blurring correction process. Specifically, the correctionstart detection section 702 performs a feature point matching process onthe image input from the demosaicing section 312 and the image acquiredin the preceding frame and stored in the first storage section 701, andcalculates a motion amount Mv1 from the feature point motion vector. Forexample, the motion amount Mv1 is the average feature point motionvector of the entire image.

The correction start detection section 702 compares the calculatedmotion amount Mv1 with a given threshold value ThMv1 to determinewhether or not the motion of the object within the image is large. Thethreshold value ThMv1 may be a value set in advance, or may beautomatically set by the control section 320. The correction startdetection section 702 determines that the doctor closely observes theobject when the motion amount Mv1 is smaller than the threshold valueThMv1 (i.e., the motion of the object is small). In this case, thecorrection start detection section 702 transmits the image to thecoefficient calculation section 703 in order to start the blurringcorrection process. The correction start detection section 702determines that the doctor screens the object when the motion amount Mv1is equal to or larger than the threshold value ThMv1 (i.e., the motionof the object is large). In this case, the correction start detectionsection 702 transmits the image to the post-processing section 314(i.e., the blurring correction process is not performed).

Note that the doctor may input a correction start instruction using theexternal I/F section 500, and the blurring correction process may beperformed based on the input correction start instruction. In this case,a correction start signal is input from the external I/F section 500 viathe control section 320 when the correction start instruction has beeninput, and the image is transmitted to the coefficient calculationsection 703.

The coefficient calculation section 703 calculates a correctioncoefficient for correcting blurring in the depth direction based on theimage input from the correction start detection section 702 and theimage acquired in the preceding frame and stored in the first storagesection 701. Specifically, feature point information obtained by thecorrection start detection section 702 is input to the coefficientcalculation section 703. The feature point information includes featurepoint information about the image in the current frame and feature pointinformation about the image in the preceding frame. The coefficientcalculation section 703 calculates a magnification Mag from the featurepoint shape similarity.

FIG. 6A shows a polygon formed by feature points extracted from an imageacquired at a time t−1, and FIG. 6B shows a polygon formed by featurepoints extracted from an image acquired at a time t. The image at thetime t is an image in the current frame, and the image at the time t−1is an image in the preceding frame stored in the first storage section701. The coefficient calculation section 703 calculates themagnification Mag so that the area of the polygon at the time t isalmost equal to the area of the polygon at the time t−1. For example,the coefficient calculation section 703 calculates the ratio of the areaof the polygon at the time t to the area of the polygon at the time t−1as the magnification Mag. The calculated magnification Mag istransmitted to the depth-direction blurring correction section 704.

The depth-direction blurring correction section 704 enlarges or reducesthe image based on the magnification Mag input from the coefficientcalculation section 703. Specifically, the depth-direction blurringcorrection section 704 performs the enlargement process by electroniczoom using the reciprocal of the magnification Mag as the enlargementfactor. For example, a known interpolation process is used for theenlargement process or the reduction process. FIG. 6C shows an exampleof an image obtained by enlarging the image acquired at the time t bythe magnification Mag. Specifically, the depth-direction blurringcorrection section 704 causes the size of the object within the imageacquired at each time (each frame) to be constant. The image subjectedto the enlargement process is transmitted to the trimming section 705.

The trimming section 705 trims the image input from the depth-directionblurring correction section 704. The trimming range may be a given rangedesignated (specified) in advance, or may be designated (specified) viathe external I/F section 500 and the control section 320. FIG. 6D showsan example of the trimming range applied to the image shown in FIG. 6C.As shown in FIG. 6E, the outer area of the image is masked when theimage is smaller than the trimming range. Note that the image may beenlarged to have the same size as that of the trimming range when theimage is smaller than the trimming range. Specifically, the trimmingsection 705 extracts an image having a constant size from the image thathas been enlarged or reduced by the depth-direction blurring correctionprocess. The trimmed image is transmitted to the second storage section706, the determination section 707, and the planar-direction blurringcorrection section 708.

The second storage section 706 stores the image input from the trimmingsection 705.

The determination section 707 determines whether or not thedepth-direction correction process is stable. Specifically, thedetermination section 707 detects a correlation value of the image inputfrom the trimming section 705 and the image in the preceding framestored in the second storage section 706. The determination section 707determines whether or not the depth-direction correction process isstable based on the correlation value.

FIG. 6F shows an example of an image obtained by trimming the imageshown in FIG. 6D. FIG. 6G shows an example of an image obtained bytrimming the image acquired in the preceding frame of the image shown inFIG. 6F. The determination section 707 performs a feature point matchingprocess on these images, and calculates a motion amount Mv2 from thefeature point motion vector. The motion amount Mv2 is the averagefeature point motion vector of the entire image.

The determination section 707 compares the calculated motion amount Mv2with a given threshold value ThMv2 to determine whether or not themotion of the object within the image is large. The threshold valueThMv2 may be a value set in advance, or may be automatically set by thecontrol section 320. The determination section 707 determines that thedepth-direction correction process is stable when the motion amount Mv2is smaller than the threshold value ThMv2 (i.e., the motion of theobject is small). In this case, the determination section 707 transmitsthe image to the planar-direction blurring correction section 708 inorder to start a planar-direction blurring correction process (i.e.,upward/downward/rightward/leftward-direction blurring correctionprocess). The determination section 707 determines that thedepth-direction correction process is not stable when the motion amountMv2 is equal to or larger than the threshold value ThMv2 (i.e., themotion of the object is large). In this case, the determination section707 transmits the image to the post-processing section 314 (i.e., theplanar-direction blurring correction process is not performed).

The planar-direction blurring correction section 708 performs theplanar-direction blurring correction process (i.e.,upward/downward/rightward/leftward-direction blurring correctionprocess) on the image input from the determination section 707. Forexample, the planar-direction blurring correction process is performedby a known electronic blurring correction process. The electronicblurring correction process calculates the inter-frame motion vector ofthe object by a matching process, and sets a trimming rangecorresponding to the motion vector, for example. Since the image istrimmed corresponding to the motion vector of the object, a trimmedimage in which blurring of the object in the planar direction issuppressed is acquired. The image subjected to the planar-directionblurring correction process is transmitted to the post-processingsection 314.

Although an example in which the determination section 707 calculatesthe feature point motion vector by the matching process has beendescribed above, another method may also be employed. For example, thedetermination section 707 may calculate the feature point motion vectorwithin the image subjected to the depth-direction blurring correctionprocess based on the feature point information obtained by thecorrection start detection section 702 and the correction coefficientcalculated by the coefficient calculation section 703.

FIG. 7 shows an example of feature point data in the blurring correctionprocess. As shown in FIG. 7, the coordinates of feature points P1 to P3within an image f(t) in the current frame and the coordinates of featurepoints P1′ to P3′ within an image f(t−1) in the preceding frame arecalculated. The feature point is an area (e.g., a lesion area or a bloodvessel intersection area) suitable for the matching process (see FIG.6A). The feature points P1′ to P3′ are linked to the feature points P1to P3 by the matching process.

The correction start detection section 702 calculates motion vectorsP1-P1′ to P3-P3′ using the coordinates of the feature points, anddetermines whether or not to start the blurring correction process basedon the calculated motion vectors. When the correction start conditionhas been satisfied, the coefficient calculation section 703 calculatesthe area of a polygon formed by the feature points P1 to P3 and the areaof a polygon formed by the feature points P1′ to P3′, and calculates themagnification Mag from the calculated areas. The depth-directionblurring correction section 704 performs the electronic zoom process onthe image f(t) using the calculated magnification Mag.

The determination section 707 calculates coordinates Mag·P1 to Mag·P3 ofthe feature points after the depth-direction blurring correction processbased on the coordinates of the feature points P1 to P3 and themagnification Mag. The determination section 707 calculates motionvectors Mag·P1-P1′ to Mag·P3-P3′ after the depth-direction blurringcorrection process based on the coordinates Mag·P1 to Mag·P3 and thecoordinates of the feature points P1′ to P3′. The determination section707 determines whether or not to start the planar-direction blurringcorrection process based on the motion vectors Mag·P1-P1′ to Mag·P3-P3′.

5. Program

In one embodiment of the invention, some or all of the processesperformed by each section of the image processing section 310 may beimplemented by software. In this case, a CPU of a computer system(described later with reference to FIG. 17, for example) executes animage processing program.

FIG. 8 shows an example of a flowchart of the image processing program.As shown in FIG. 8, header information (e.g., light sourcesynchronization signal and imaging (photographing) mode) is input to thetime-series images (S11).

The images are input to an image buffer allocated in advance (S12). TheOB clamp process, the gain control process, the WB correction process,and the like are performed on the input images (S13). The demosaicingprocess is performed on the input time-series images according to thelight source synchronization signal (S14). Next, the blurring correctionprocess is performed on each image (S15). The blurring correctionprocess is described in detail later with reference to FIG. 9. Thegrayscale, conversion process, the color process, the contourenhancement process, and the like are performed on the image subjectedto the blurring correction process (S16). The post-processed image isthen output (S17).

Whether or not the final image among the time-series images has beenprocessed is determined (S18). When it has been determined that thefinal image has not been processed (S18, No), the processes in the stepsS12 to S17 are performed on the subsequent image. When it has beendetermined that the final image has been processed (S18, Yes), theprocess is terminated.

FIG. 9 shows an example of a flowchart of the blurring correctionprocess (step S15 in FIG. 8). As shown in FIG. 9, the feature pointmatching process is performed on the input image and the previous(preceding) image, and the motion amount Mv1 is calculated from thefeature point motion vector (S21). An image stored in a first work imagebuffer used in a step S22 is used as the previous image. The input imageis copied into the first work image buffer allocated in advance (S22).

Next, whether or not the motion amount Mv1 is smaller than the thresholdvalue ThMv1 set in advance is determined (S23). When the motion amountMv1 is equal to or larger than the threshold value ThMv1 (S23, No), theprocess is terminated. When the motion amount Mv1 is smaller than thethreshold value ThMv1 (S23, Yes), the magnification Mag is calculated sothat the area of a polygon formed by the feature points included in theimage stored in the first work buffer is almost equal to the area of apolygon formed by the feature points included in the input image (S24).The depth-direction blurring correction process is performed byperforming the enlargement process using the magnification Mag (S25).The image subjected to the depth-direction blurring correction processis trimmed using a given range (S26). The feature point matching processis performed on the trimmed image and the previous trimmed image, andthe motion amount Mv2 is calculated from the feature point motion vector(S27). An image stored in a second work image buffer used in a step S28is used as the previous trimmed image. The trimmed image is copied intothe second work image buffer allocated in advance (S28).

Next, whether or not the motion amount Mv2 is smaller than the thresholdvalue ThMv2 set in advance is determined (S29). When the motion amountMv2 is equal to or larger than the threshold value ThMv2 (S29, No), theprocess is terminated. When the motion amount Mv2 is smaller than thethreshold value ThMv2, the planar-direction blurring correction process(upward/downward/rightward/leftward-direction blurring correctionprocess) is performed (S30).

This makes it possible to display the object to have a constant sizeeven when the object moves due to the motion of the internal organscaused by the heartbeat. Therefore, it is possible to provide anendoscope apparatus that improves the lesion area observation capabilitywhile reducing the burden on the doctor.

The object may be blurred in the depth direction of the imaging sectiondue to the motion of the internal organs caused by the heartbeat, sothat the visibility of the observed area may deteriorate. For example,the amount of blurring in the depth direction increases duringmagnifying observation since the object is observed at a highmagnification in a state in which the imaging section is positionedright in front of the object.

The endoscope apparatus according to the first configuration exampleincludes an image acquisition section, the coefficient calculationsection 703, and the depth-direction blurring correction section 704(see FIG. 5). The image acquisition section acquires images in timeseries. The coefficient calculation section 703 calculates a correctioncoefficient Mag for correcting blurring in the depth direction that is adirection along the optical axis of the imaging section 200. Thedepth-direction blurring correction section 704 performs a process thatcorrects blurring in the depth direction on the images acquired in timeseries based on the correction coefficient Mag.

This makes it possible to suppress blurring of a moving image in thedepth direction. Specifically, blurring of the moving image can bereduced even when the object moves in the depth direction due to themotion of the internal organs by correcting blurring of the image in thedepth direction based on the correction coefficient Mag. This makes itpossible to improve the observation capability (e.g., visibility), andreduce the burden on the doctor.

In the first configuration example, the demosaicing section 312corresponds to the image acquisition section (see FIG. 4). Specifically,the imaging element 206 captures images in time series as a movingimage, the A/D conversion section 207 converts the images into digitaldata, and the preprocessing section 311 performs the preprocess on thedigital data. The demosaicing section 312 performs the demosaicingprocess to acquire images in time series as a moving image.

The image acquisition section acquires a first image and a second imagesubsequent to the first image in time series. The coefficientcalculation section 703 calculates the correction coefficient Mag thatcorresponds to the magnification of the second image with respect to thefirst image. The depth-direction blurring correction section 704performs the depth-direction blurring correction process by correcting achange in magnification of images due to blurring in the depth directionbased on the correction coefficient Mag.

This makes it possible to correct blurring in the depth direction basedon the correction coefficient Mag. Specifically, a change inmagnification of images acquired in time series can be corrected basedon the correction coefficient Mag by calculating the correctioncoefficient Mag that corresponds to the magnification of the secondimage with respect to the first image.

The first image refers to an image in the preceding frame of theprocessing target frame, and the second image refers to an image in theprocessing target frame, for example. The correction coefficient thatcorresponds to the magnification of the second image with respect to thefirst image may be magnification calculated from the first image and thesecond image by image processing, or may be magnification calculatedfrom another information (e.g., the position of an AF lens (describedlater). A change in magnification is not necessarily corrected bycorrecting the magnification of the second image. For example, theimaging magnification of a third image subsequent to the second imagemay be corrected by optical zoom (described later).

The endoscope apparatus according to the first configuration exampleincludes the correction start detection section 702 that detects a starttiming of the depth-direction blurring correction process (see FIG. 5).The depth-direction blurring correction section 704 starts thedepth-direction blurring correction process when the correction startdetection section 702 has detected the start timing of thedepth-direction blurring correction process.

This makes it possible to perform the depth-direction blurringcorrection process when the depth-direction blurring correction processhas become necessary. For example, the depth-direction blurringcorrection process can be started when it has been determined that theamount of blurring of the object image in the direction along theoptical axis of the imaging section has exceeded a given referencerange. This makes it possible to prevent a situation in which thedepth-direction blurring correction process hinders a lesion area searchprocess that is performed while moving the imaging section along thedigestive tract, for example.

The endoscope apparatus according to the first configuration exampleincludes the determination section 707 and the planar-direction blurringcorrection section 708 (see FIG. 5). The determination section 707determines whether or not the amount of blurring in the depth directionafter the depth-direction blurring correction process is within a givenreference range. The planar-direction blurring correction section 708that performs a planar-direction blurring correction process thatcorrects planar-direction blurring that is blurring in a direction thatperpendicularly intersects the optical axis of the imaging section 200when the determination section 707 has determined that the amount ofblurring in the depth direction is within the given reference range.

This makes it possible to correct planar-direction blurring when theamount of blurring in the depth direction is within the given referencerange (i.e., is stable). For example, the magnification of the imageschanges to a large extent in a state in which blurring in the depthdirection has not been sufficiently corrected. In this case, theplanar-direction blurring correction accuracy deteriorates due to poorinter-frame matching accuracy. Therefore, the planar-direction blurringcorrection accuracy can be improved by determining blurring in the depthdirection, and then correcting planar-direction blurring.

In the first configuration example, the coefficient calculation section703 calculates the magnification of the second image with respect to thefirst image as the correction coefficient Mag based on the ratio of thearea of a region enclosed by the feature points P1′ to P3′ included inthe first image and the area of a region enclosed by the feature pointsP1 to P3 included in the second image corresponding to the featurepoints P1′ to P3′ included in the first image, as described above withreference to FIG. 6A and the like.

This makes it possible to calculate the magnification of the secondimage with respect to the first image as the correction coefficient.Moreover, the correction coefficient can be calculated by imageprocessing.

In the first configuration example, the depth-direction blurringcorrection section 704 performs the depth-direction blurring correctionprocess by enlarging or reducing the image based on the correctioncoefficient Mag. Specifically, the coefficient calculation section 703calculates the magnification of the second image with respect to thefirst image as the correction coefficient Mag. The depth-directionblurring correction section 704 enlarges or reduces the second imagebased on the correction coefficient Mag. The endoscope apparatusincludes the trimming section that trims an image having a given sizefrom the image subjected to the depth-direction blurring correctionprocess (see FIG. 5).

This makes it possible to perform the depth-direction blurringcorrection process by electronic zoom that increases or reduces theimage size. Specifically, blurring in the depth direction be correctedby calculating the magnification as the correction coefficient, andenlarging the second image by electronic zoom using the calculatedmagnification. Since the depth-direction blurring correction process canbe performed without using a mechanical mechanism by utilizingelectronic zoom, it is unnecessary to increase the thickness of the endof the imaging section. Moreover, the image can be displayed to have agiven size by trimming the image even if the image size has changed dueto electronic zoom.

In the first configuration example, the correction start detectionsection 702 determines whether or not to start the depth-directionblurring correction process based on the motion information about theobject included in the image. The depth-direction blurring correctionsection 704 starts the depth-direction blurring correction process whenthe correction start detection section 702 has determined to start thedepth-direction blurring correction process. Specifically, thecorrection start detection section 702 calculates the motion amount Mv1that indicates the amount of blurring in the depth direction based onthe motion information about the object included in the first image andthe second image, and determines to start the depth-direction blurringcorrection process when the motion amount Mv1 is equal to or larger thanthe threshold value ThMv1.

For example, the correction start detection section 702 calculates thecoordinates of the feature points P1 to P3 by the matching process,calculates the motion vectors of the feature points P1 to P3 as themotion information, and calculates the average value of the motionvectors as the motion amount Mv1.

This makes it possible to detect the start timing of the depth-directionblurring correction process based on the motion information calculatedby image processing. It is possible to determine whether or not theamount of blurring in the depth direction is within a given referencerange by determining whether or not the motion amount Mv1 is smallerthan the threshold value ThMv1.

The determination section 707 determines whether or not the amount ofblurring in the depth direction after the depth-direction blurringcorrection process is within a given reference range based on the motioninformation about the image after the depth-direction blurringcorrection process. Specifically, the determination section 707calculates the motion amount Mv2 that indicates the amount of blurringin the depth direction based on the motion information about the featurepoints P1′ to P3′ included in the first image and the feature points P1to P3 included in the second image corresponding to the feature pointsP1′ to P3′ included in the first image after the depth-directionblurring correction process. The determination section 707 determines tostart the planar-direction blurring correction process when the motionamount Mv2 is smaller than the threshold value ThMv2.

This makes it possible to detect the start timing of theplanar-direction blurring correction process based on the motioninformation about the image after the depth-direction blurringcorrection process. It is possible to determine whether or not theamount of blurring in the depth direction after the depth-directionblurring correction process is within a given reference range bydetermining whether or not the motion amount Mv2 is smaller than thethreshold value ThMv2.

In the first configuration example, the planar-direction blurringcorrection section 708 performs the planar-direction blurring correctionprocess by an electronic blurring correction process. The electronicblurring correction process may be performed using the feature pointmotion vector calculated by the determination section 707, or may beperformed using a motion vector calculated by another matching process.

Although an example in which the motion information about the object iscalculated based on the captured image has been described above, anotherconfiguration may also be employed. For example, a sensor may beprovided on the end of the imaging section 200, and motion informationabout the end of the imaging section 200 may be acquired using thesensor. The depth-direction blurring correction process or theplanar-direction blurring correction process may be performed based onthe acquired motion information.

6. Second Configuration Example of Endoscope Apparatus

In one embodiment of the invention, the imaging section may perform anautofocus process, and the magnification Mag may be calculated based onthe in-focus object plane set by the autofocus process. FIG. 10 shows asecond configuration example of the endoscope apparatus employed whenimplementing such a feature. The endoscope apparatus shown in FIG. 10includes a light source section 100, an imaging section 200, a controldevice 300, a display section 400, and an external I/F section 500. Notethat the same elements as those described above in connection with thefirst configuration example (see FIG. 1 and the like) are indicated bythe same reference symbols. Description of these elements isappropriately omitted.

The imaging section 200 includes an AF lens 208 and an AF lens driversection 209. The AF lens driver section 209 moves the AF lens 208 in theoptical axis direction. The AF lens driver section 209 moves the AF lens208 to a given position based on a control signal output from thecontrol section 320.

The control device 300 according to the second configuration examplediffers from the control device 300 according to the first configurationexample as to the specific configuration of the image processing section310, and the control process of the control section 320 is changed toimplement an AF control process.

FIG. 11 shows a second specific configuration example of the imageprocessing section 310. The image processing section 310 additionallyincludes an AF control section 315. The details of the blurringcorrection section 313 also differ from those of the blurring correctionsection 313 according to the first configuration example (see FIG. 4 andthe like). The demosaicing section 312 is connected to the blurringcorrection section 313 and the AF control section 315. The AF controlsection 315 is connected to the blurring correction section 313. The AFcontrol section 315 is bidirectionally connected to the control section320.

The AF control section 315 performs the AF control process under controlof the control section 320. For example, the AF control section 315performs the AF control process when the variable aperture 204 is openedunder control of the control section 320. Specifically, the AF controlsection 315 performs a contrast AF process based on the image input fromthe demosaicing section 312, and transmits an AF control process signalto the AF lens driver section 209 via the control section 320. Thecontrast AF process is implemented by a known process, for example. Anoutline of the contrast AF process is described below.

Specifically, an AF evaluation value A1 is calculated at an initialposition p1 of the lens. The sum of the signal values of images passedthrough a high-pass filter is used as the AF evaluation value, forexample. The AF lens 208 is moved to a position p2 at a given distancefrom the position p1 in the infinite direction or the direction oppositeto the infinite direction, and an AF evaluation value A2 is calculatedat the position p2. The AF evaluation values A1 and A2 are compared todetermine the direction of the focus target position (the objectposition). The AF lens 208 is moved by a given distance toward the focustarget position, and AF evaluation values A3 and A4 are calculated atpositions p3 and p4, respectively. When the AF evaluation value haspassed through the peak (i.e., the focus target position), the focustarget position is calculated by interpolation calculations using threepoints around the focus target position and the AF evaluation values.Linear interpolation or the like is used for the interpolationcalculations. Note that Lagrange interpolation, spline interpolation, orthe like may also be used. The AF lens is moved to the positioncorresponding to the calculated focus target position so that thein-focus object plane position is moved to the focus target position,and the AF process is repeated.

The details of the blurring correction section 313 are described below.FIG. 12 shows a specific configuration example of the blurringcorrection section 313. The blurring correction section 313 according tothe second configuration example differs from the blurring correctionsection 313 according to the first configuration example (see FIG. 5 andthe like) in that the first storage section 701 is omitted. The detailsof the correction start detection section 702 and the coefficientcalculation section 703 also differ from those according to the firstconfiguration example. The AF control section 315 is connected to thecorrection start detection section 702 and the coefficient calculationsection 703.

The correction start detection section 702 starts the blurringcorrection process at a timing at which the AF control section 315 hasstarted the AF control process. The correction start detection section702 transmits the image to the coefficient calculation section 703 whenthe AF control section 315 performs the AF control process. Thecorrection start detection section 702 transmits the image to thepost-processing section 314 when the AF control section 315 does notperform the AF control process.

The coefficient calculation section 703 calculates a depth-directionblurring correction coefficient based on an AF control signal input fromthe AF control section 315. Specifically, the coefficient calculationsection 703 calculates the magnification Mag based on the position ofthe AF lens 208 and the position of the AF lens 208 when the in-focusobject plane has been moved at the next timing. More specifically, thecoefficient calculation section 703 calculates a in-focus object planedistance fa at the position of the AF lens 208 and a in-focus objectplane distance fb at the position of the AF lens 208 when the in-focusobject plane position has been moved at the next timing using a look-uptable (LUT) that indicates the relationship between the position of theAF lens 208 and the in-focus object plane distance. The coefficientcalculation section 703 calculates the magnification Mag from the ratioof the in-focus object plane distance fa and the in-focus object planedistance fb.

For example, the LUT is configured so that the position of the AF lens208 is linked to the in-focus object plane distance as shown below. Notethat n is a natural number.

Lens position: in-focus object plane distance f

-   -   x1: fx1    -   x2: fx2    -   x3: fx3    -   xn: fxn

The in-focus object plane distance f is determined from the position ofthe AF lens by acquiring AF lens position information from the AFcontrol section 315, and referring to the LUT. When the in-focus objectplane distance when the processing target image is captured is referredto as ft, and the in-focus object plane distance when the image in thepreceding frame of the processing target image is captured is referredto as ft−1, the magnification Mag is calculated by the followingexpression (1).Mag=ft−1/ft  (1)

7. Second Example of Program

In one embodiment of the invention, some or all of the processesperformed by each section of the image processing section 310 shown inFIG. 11 may be implemented by software. In this case, a CPU of acomputer system (described later with reference to FIG. 17, for example)executes an image processing program.

FIG. 13 shows a second example of a flowchart of the blurring correctionprocess. Note that the flowchart shown in FIG. 13 differs from theflowchart shown in FIG. 8 as to the details of the steps S11 and S15.The remaining steps are the same as those shown in FIG. 8. Specifically,the AF control signal is added to the header information in the stepS11. The details of the step S15 are described below with reference toFIG. 13.

The flowchart shown in FIG. 13 differs from the flowchart shown in FIG.9 in that the steps S21 to S23 are omitted, and a step S123 isadditionally provided. In the step S123, whether or not the image is animage acquired when the AF control process is performed is determinedfrom the header information about the image. When the image is an imageacquired when the AF control process is performed, the processtransitions to the step S24. When the image is an image acquired whenthe AF control process is not performed, the process ends.

According to the second configuration example, the imaging section 200includes an optical system that performs the autofocus process. Thecoefficient calculation section 703 calculates the correctioncoefficient Mag based on the in-focus object plane position (thein-focus object plane distance) of the optical system adjusted by theautofocus process. Specifically, the coefficient calculation section 703calculates the magnification ft−1/ft of the second image with respect tothe first image as the correction coefficient Mag based on the in-focusobject plane ft−1 when the first image is captured and the in-focusobject plane ft when the second image is captured.

According to this configuration, an in-focus state can always beachieved even during magnifying observation (i.e., the depth of field ofthe imaging system is shallow) by utilizing the autofocus process. Thismakes it possible to necessarily perform the depth-direction blurringcorrection process in an in-focus state, so that the burden on thedoctor can be reduced while improving the observation capability.Moreover, since the distance from the imaging section to the object canbe determined by the in-focus object plane, the magnification can becalculated based on the in-focus object plane ratio.

The autofocus process is implemented by causing the control section 320to control the AF lens driver section 209 so that the AF lens driversection 209 drives the AF lens 208 (see FIG. 10). The in-focus objectplane of the optical system refers to the distance from the imagingsection to the object when the object is in focus, and is determined bythe position of the AF lens 208. The position of the AF lens refers tothe distance from the AF lens 208 to the objective lens 203, forexample. The in-focus object plane position (in-focus object planedistance f) is calculated from the position of the AF lens.

In the second configuration example, the correction start detectionsection 702 determines whether or not to start the depth-directionblurring correction process based on the state of the autofocus process.For example, the optical system of the imaging section 200 may performan optical zoom process. In this case, the autofocus process is enabledin a magnifying observation mode in which the magnification is higherthan the optical zoom magnification employed in a normal observationmode. The correction start detection section 702 determines to start thedepth-direction blurring correction process when the autofocus processhas been enabled.

This makes it possible to detect the start timing of the depth-directionblurring correction process based on the state of the autofocus process.For example, the start timing can be detected corresponding to theenabled/disabled state of the autofocus process, the focus adjustmentfrequency, the moving amount of the position of the AF lens, or thelike. Moreover, the depth-direction blurring correction process can beperformed in the magnifying observation mode in which the amount ofblurring is large, by enabling the autofocus process in the magnifyingobservation mode to detect the start timing.

8. Third Configuration Example of Endoscope Apparatus

In one embodiment of the invention, the imaging section may have anoptical zoom function, and blurring in the depth direction may becorrected by optical zoom. FIG. 14 shows a third configuration exampleof the endoscope apparatus employed when implementing such a feature.The endoscope system includes a light source section 100, an imagingsection 200, a control device 300, a display section 400, and anexternal I/F section 500. Note that the same elements as those describedabove in connection with the first configuration example (see FIG. 1 andthe like) are indicated by the same reference symbols. Description ofthese elements is appropriately omitted.

The imaging section 200 includes a zoom lens 210 and a zoom lens driversection 211. The zoom lens driver section 211 moves the zoom lens 210 inthe optical axis direction. The zoom lens driver section 211 moves thezoom lens 210 to a given position based on a control signal output fromthe control section 320.

The control device 300 according to the third configuration examplediffers from the control device 300 according to the first configurationexample (see FIG. 1) as to the details of the image processing section310, and the control process of the control section 320 is changed toimplement a zoom control process.

FIG. 15 shows a third specific configuration example of the imageprocessing section 310. The image processing section 310 according tothe third configuration example differs from the image processingsection 310 according to the first configuration example (see FIG. 4) inthat a zoom control section 316 is additionally provided. The details ofthe blurring correction section 313 also differ from those of theblurring correction section 313 according to the first configurationexample. The blurring correction section 313 is connected to the zoomcontrol section 316. The zoom control section 316 is bidirectionallyconnected to the control section 320.

The zoom control section 316 performs an optical zoom control processbased on the correction coefficient used when performing thedepth-direction blurring correction process. Specifically, the zoomcontrol section 316 transmits a zoom control signal to the zoom lensdriver section 211 via the control section 320 based on themagnification Mag input from the blurring correction section 313. Thezoom control section 316 calculates a focal length fb after the zoomprocess from the magnification Mag and the current focal length fa. Thezoom control section 316 calculates the lens position from the focallength fb after the zoom process. For example, the zoom control section316 calculates the lens position referring to a LUT in which the focallength is linked to the lens position, and controls the zoom lens driversection 211 based on the lens position.

The details of the blurring correction section 313 are described below.FIG. 16 shows a third specific configuration example of the blurringcorrection section 313. The blurring correction section 313 shown inFIG. 16 differs from the blurring correction section 313 according tothe first configuration example (see FIG. 5) in that the depth-directionblurring correction section 704, the trimming section 705, the secondstorage section 706, and the determination section 707 are omitted. Thedetails of the correction start detection section 702 and thecoefficient calculation section 703 also differ from those according tothe first configuration example. The coefficient calculation section 703is connected to the planar-direction blurring correction section 708,the post-processing section 314, and the zoom control section 316.

The correction start detection section 702 determines whether or not tostart the depth-direction blurring correction process (optical zoomcontrol process) or the planar-direction blurring correction process.The correction start detection section 702 calculates the motion amountMv by the matching process in the same manner as described above (seeFIG. 6A and the like). The correction start detection section 702compares the motion amount Mv with threshold values ThMv1 and ThMv2(ThMv1>ThMv2).

When the motion amount Mv is smaller than the threshold value ThMv2, thecorrection start detection section 702 transmits a control signal thatinstructs start of the planar-direction blurring correction process tothe coefficient calculation section 703 via the control section 320, andtransmits the image to the coefficient calculation section 703. When themotion amount Mv is smaller than the threshold value ThMv1, and is equalto or larger than the threshold value ThMv2, the correction startdetection section 702 transmits a control signal that does not instructstart of the planar-direction blurring correction process to thecoefficient calculation section 703 via the control section 320, andtransmits the image to the coefficient calculation section 703. When themotion amount My is equal to or larger than the threshold value ThMv1,the correction start detection section 702 transmits the image to thepost-processing section 314.

The coefficient calculation section 703 transmits the calculatedmagnification Mag to the zoom control section 316. When the controlsignal input from the control section 320 instructs start of theplanar-direction blurring correction process, the coefficientcalculation section 703 transmits the image to the planar-directionblurring correction section 708. When the control signal input from thecontrol section 320 does not instruct start of the planar-directionblurring correction process, the coefficient calculation section 703transmits the image to the post-processing section 314.

According to the third configuration example, the imaging section 200includes an optical system that performs the optical zoom process. Thedepth-direction blurring correction section 704 performs thedepth-direction blurring correction process by adjusting the opticalzoom magnification of the optical system based on the correctioncoefficient Mag. For example, a first image, second image, and a thirdimage are acquired in time series, and the magnification of the secondimage with respect to the first image is used as the correctioncoefficient. The optical zoom magnification is changed by the reciprocalof the correction coefficient when capturing the third image subsequentto the second image.

It is possible to necessarily perform the depth-direction blurringcorrection process without causing a deterioration in image by utilizingthe optical zoom process, so that the burden on the doctor can bereduced while improving the observation capability. The number ofheartbeats per second is about one, and is smaller than a normal imagingframe rate (e.g., 30 frames per second). Therefore, the correctionprocess is performed with a delay of one frame when using the opticalzoom process. However, blurring in the depth direction can besufficiently suppressed.

Although an example in which the imaging element 206 is a singlemonochrome imaging element has been described above, anotherconfiguration may also be employed. For example, the imaging element 206may be an imaging element using a primary color Bayer array filter, ormay be an imaging element using a complementary color filter. In thiscase, the rotary color filter 104 and the rotation driver section 105included in the light source section can be omitted. The demosaicingsection 312 included in the image processing section 310 performs aninterpolation process (demosaicing process). The interpolation processmay be implemented by known linear interpolation or the like.

Although an example in which the imaging section has the AF function orthe optical zoom function has been described above, the imaging sectionmay have both the AF function and the optical zoom function.

9. Computer System

Although an example in which each section of the image processingsection 310 is implemented by hardware has been described above, anotherconfiguration may also be employed. For example, a CPU may perform theprocess of each section on an image acquired using an imaging apparatussuch as a capsule endoscope. Specifically, the process of each sectionmay be implemented by software by causing the CPU to execute a program.Alternatively, part of the process of each section may be implemented bymeans of software. The image acquired in advance refers to a Bayer arrayoutput image output from the A/D conversion section 207 and recorded ona recording medium as a RAW file, for example.

When separately providing the imaging section, and implementing theprocess of each section of the image processing section 310 by means ofsoftware, a known computer system (e.g., work station or personalcomputer) may be used as an image processing device. A program (imageprocessing program) that implements the process of each section of theimage processing section 310 may be provided in advance, and executed bythe CPU of the computer system.

FIG. 17 is a system configuration diagram showing the configuration of acomputer system 600 according to a modification. FIG. 18 is a blockdiagram showing the configuration of a main body 610 of the computersystem 600. As shown in FIG. 17, the computer system 600 includes themain body 610, a display 620 that displays information (e.g., image) ona display screen 621 based on instructions from the main body 610, akeyboard 630 that allows the user to input information to the computersystem 600, and a mouse 640 that allows the user to designate anarbitrary position on the display screen 621 of the display 620.

As shown in FIG. 18, the main body 610 of the computer system 600includes a CPU 611, a RAM 612, a ROM 613, a hard disk drive (HDD) 614, aCD-ROM drive 615 that receives a CD-ROM 660, a USB port 616 to which aUSB memory 670 is removably connected, an I/O interface 617 thatconnects the display 620, the keyboard 630, and the mouse 640, and a LANinterface 618 that is used to connect to a local area network or a widearea network (LAN/WAN)N1.

The computer system 600 is connected to a modem 650 that is used toconnect to a public line N3 (e.g., Internet). The computer system 600 isalso connected to a personal computer (PC) 681 (i.e., another computersystem), a server 682, a printer 683, and the like via the LAN interface618 and the local area network or the large area network N1.

The computer system 600 implements the functions of the image processingdevice by reading an image processing program (e.g., an image processingprogram that implements the process described with reference to FIGS. 8,9, and 13) recorded on a given recording medium, and executing the imageprocessing program. The given recording medium may be an arbitraryrecording medium that records the image processing program that can beread by the computer system 600, such as the CD-ROM 660, the USB memory670, a portable physical medium (e.g., MO disk, DVD disk, flexible disk(FD), magnetooptical disk, or IC card), a stationary physical medium(e.g., HDD 614, RAM 612, or ROM 613) that is provided inside or outsidethe computer system 600, or a communication medium that temporarilystores a program during transmission (e.g., the public line N3 connectedvia the modem 650, or the local area network or the wide area network N1to which the computer system (PC) 681 or the server 682 is connected).

Specifically, the image processing program is recorded on a recordingmedium (e.g., portable physical medium, stationary physical medium, orcommunication medium) so that the image processing program can be readby a computer. The computer system 600 implements the functions of theimage processing device by reading the image processing program fromsuch a recording medium, and executing the image processing program.Note that the image processing program need not necessarily be executedby the computer system 600. The invention may be similarly applied tothe case where the computer system (PC) 681 or the server 682 executesthe image processing program, or the computer system (PC) 681 and theserver 682 execute the image processing program in cooperation.

This makes it possible to store image data, and process the stored imagedata by means of software using a computer system (e.g., PC) (e.g.,capsule endoscope).

The above embodiments may also be applied to a computer program productthat stores a program code that implements each section (e.g.,preprocessing section, demosaicing section, blurring correction section,and post-processing section) according to the above embodiments.

The program code implements an image acquisition section that acquiresimages in time series, a coefficient calculation section that calculatesa correction coefficient for correcting blurring in a depth directionthat is a direction along an optical axis of an imaging section, and adepth-direction blurring correction section that performs a process thatcorrects blurring in the depth direction on the images based on thecorrection coefficient.

The term “computer program product” refers to an information storagemedium, a device, an instrument, a system, or the like that stores aprogram code, such as an information storage medium (e.g., optical diskmedium (e.g., DVD), hard disk medium, and memory medium) that stores aprogram code, a computer that stores a program code, or an Internetsystem (e.g., a system including a server and a client terminal), forexample. In this case, each element and each process according to theabove embodiments are implemented by corresponding modules, and aprogram code that includes these modules is recorded in the computerprogram product.

The embodiments according to the invention and modifications thereofhave been described above. Note that the invention is not limited to theabove embodiments and modifications thereof. Various modifications andvariations may be made without departing from the scope of theinvention. A plurality of elements disclosed in connection with theabove embodiments and modifications thereof may be appropriatelycombined. For example, some of the elements disclosed in connection withthe above embodiments and modifications thereof may be omitted. Some ofthe elements disclosed in connection with the above embodiments andmodifications thereof may be appropriately combined. Specifically,various modifications and applications are possible without materiallydeparting from the novel teachings and advantages of the invention.

Any term (e.g., image signal, endoscope apparatus, or optical axisdirection) cited with a different term (e.g., image, endoscope system,or depth direction) having a broader meaning or the same meaning atleast once in the specification and the drawings may be replaced by thedifferent term in any place in the specification and the drawings.

What is claimed is:
 1. An endoscope apparatus comprising: an imageacquisition section that acquires images in time series; a coefficientcalculation section that calculates a correction coefficient forcorrecting blurring in a depth direction that is a direction along anoptical axis of an imaging section; and a depth-direction blurringcorrection section that performs a depth-direction blurring correctionprocess on the images acquired in time series based on the correctioncoefficient, the depth-direction blurring correction process being aprocess that corrects blurring in the depth direction; wherein the imageacquisition section acquiring a first image and a second image in timeseries, the second image being subsequent to the first image, thecoefficient calculation section calculating the correction coefficientthat corresponds to a magnification of the second image with respect tothe first image, and the depth-direction blurring correction sectionperforming the depth-direction blurring correction process by correctinga change in magnification of the images due to blurring in the depthdirection based on the correction coefficient.
 2. The endoscopeapparatus as defined in claim 1, further comprising: a correction startdetection section that detects a start timing of the depth-directionblurring correction process, the depth-direction blurring correctionsection starting the depth-direction blurring correction process whenthe correction start detection section has detected the start timing. 3.The endoscope apparatus as defined in claim 2, further comprising: adetermination section that determines whether or not an amount ofblurring in the depth direction after the depth-direction blurringcorrection process is within a given reference range; and aplanar-direction blurring correction section that performs aplanar-direction blurring correction process when the determinationsection has determined that the amount of blurring in the depthdirection is within the given reference range, the planar-directionblurring correction process being a process that correctsplanar-direction blurring that is blurring in a direction thatperpendicularly intersects the optical axis.
 4. An endoscope apparatuscomprising: an image acquisition section that acquires images in timeseries; a coefficient calculation section that calculates a correctioncoefficient for correcting blurring in a depth direction that is adirection along an optical axis of an imaging section; and adepth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction; wherein the image acquisition section acquiring afirst image and a second image in time series, the second image beingsubsequent to the first image, and the coefficient calculation sectioncalculating a magnification of the second image with respect to thefirst image as the correction coefficient based on a ratio of an area ofa region enclosed by feature points included in the first image and anarea of a region enclosed by feature points included in the secondimage, the feature points included in the second image corresponding tothe feature points included in the first image.
 5. An endoscopeapparatus comprising: an image acquisition section that acquires imagesin time series; a coefficient calculation section that calculates acorrection coefficient for correcting blurring in a depth direction thatis a direction along an optical axis of an imaging section; and adepth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction; wherein the imaging section including an opticalsystem that performs an autofocus process, the coefficient calculationsection calculating the correction coefficient based on an in-focusobject plane position of the optical system adjusted by the autofocusprocess; the image acquisition section acquiring a first image and asecond image in time series, the second image being subsequent to thefirst image, and the coefficient calculation section calculating amagnification of the second image with respect to the first image as thecorrection coefficient based on the in-focus object plane position whenthe first image is captured and the in-focus object plane position whenthe second image is captured.
 6. An endoscope apparatus comprising: animage acquisition section that acquires images in time series; acoefficient calculation section that calculates a correction coefficientfor correcting blurring in a depth direction that is a direction alongan optical axis of an imaging section; and a depth-direction blurringcorrection section that performs a depth-direction blurring correctionprocess on the images acquired in time series based on the correctioncoefficient, the depth-direction blurring correction process being aprocess that corrects blurring in the depth direction; wherein thedepth-direction blurring correction section performing thedepth-direction blurring correction process by enlarging or reducing animage among the images based on the correction coefficient; the imageacquisition section acquiring a first image and a second image in timeseries, the second image being subsequent to the first image, thecoefficient calculation section calculating a magnification of thesecond image with respect to the first image as the correctioncoefficient, and the depth-direction blurring correction sectionenlarging or reducing the second image based on the correctioncoefficient.
 7. An endoscope apparatus comprising: an image acquisitionsection that acquires images in time series; a coefficient calculationsection that calculates a correction coefficient for correcting blurringin a depth direction that is a direction along an optical axis of animaging section; a depth-direction blurring correction section thatperforms a depth-direction blurring correction process on the imagesacquired in time series based on the correction coefficient, thedepth-direction blurring correction process being a process thatcorrects blurring in the depth direction; and a trimming section thatperforms a trimming process trimming an image having a given size froman image subjected to the depth-direction blurring correction process;wherein the depth-direction blurring correction section performing thedepth-direction blurring correction process by enlarging or reducing animage among the images based on the correction coefficient.
 8. Anendoscope apparatus comprising: an image acquisition section thatacquires images in time series; a coefficient calculation section thatcalculates a correction coefficient for correcting blurring in a depthdirection that is a direction along an optical axis of an imagingsection; a depth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction; a correction start detection section that detects astart timing of the depth-direction blurring correction process, thecorrection start detection section determining whether or not to startthe depth-direction blurring correction process based on motioninformation about an object included in the images, and thedepth-direction blurring correction section starting the depth-directionblurring correction process when the correction start detection sectionhas determined to start the depth-direction blurring correction process.9. The endoscope apparatus as defined in claim 8, the image acquisitionsection acquiring a first image and a second image in time series, thesecond image being subsequent to the first image, the correction startdetection section calculating a motion amount that indicates an amountof blurring in the depth direction based on the motion information aboutthe object included in the first image and the second image, and thecorrection start detection section determining to start thedepth-direction blurring correction process when the motion amount islarger than a threshold value.
 10. An endoscope apparatus comprising: animage acquisition section that acquires images in time series; acoefficient calculation section that calculates a correction coefficientfor correcting blurring in a depth direction that is a direction alongan optical axis of an imaging section; a depth-direction blurringcorrection section that performs a depth-direction blurring correctionprocess on the images acquired in time series based on the correctioncoefficient, the depth-direction blurring correction process being aprocess that corrects blurring in the depth direction; a correctionstart detection section that detects a start timing of thedepth-direction blurring correction process, the imaging sectionincluding an optical system that performs an autofocus process, and thecorrection start detection section determining whether or not to startthe depth-direction blurring correction process based on a state of theautofocus process.
 11. The endoscope apparatus as defined in claim 10,the optical system performing an optical zoom process, the autofocusprocess being enabled in a magnifying observation mode, an optical zoommagnification in the magnifying observation mode being higher than anoptical zoom magnification in a normal observation mode, and thecorrection start detection section determining to start thedepth-direction blurring correction process when the autofocus processhas been enabled.
 12. An endoscope apparatus comprising: an imageacquisition section that acquires images in time series; a coefficientcalculation section that calculates a correction coefficient forcorrecting blurring in a depth direction that is a direction along anoptical axis of an imaging section; a depth-direction blurringcorrection section that performs a depth-direction blurring correctionprocess on the images acquired in time series based on the correctioncoefficient, the depth-direction blurring correction process being aprocess that corrects blurring in the depth direction; and a correctionstart detection section that detects a start timing of thedepth-direction blurring correction process, wherein the correctionstart detection section determining whether or not to start thedepth-direction blurring correction process based on an external input.13. An endoscope apparatus comprising: an image acquisition section thatacquires images in time series; a coefficient calculation section thatcalculates a correction coefficient for correcting blurring in a depthdirection that is a direction along an optical axis of an imagingsection; a depth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction; a determination section that determines whether ornot an amount of blurring in the depth direction after thedepth-direction blurring correction process is within a given referencerange based on motion information about the images after thedepth-direction blurring correction process; and a planar-directionblurring correction section that performs a planar-direction blurringcorrection process when the determination section has determined thatthe amount of blurring in the depth direction is within the givenreference range, the planar-direction blurring correction process beinga process that corrects planar-direction blurring that is blurring in adirection that perpendicularly intersects the optical axis.
 14. Theendoscope apparatus as defined in claim 13, the image acquisitionsection acquiring a first image and a second image in time series, thesecond image being subsequent to the first image, the determinationsection calculating a motion amount that indicates an amount of blurringin the depth direction based on the motion information after thedepth-direction blurring correction process, the motion informationbeing motion information between feature points included in the firstimage and feature points included in the second image, the featurepoints included in the second image corresponding to the feature pointsincluded in the first image, and the determination section determiningto start the planar-direction blurring correction process when themotion amount is smaller than a threshold value.
 15. The endoscopeapparatus as defined in claim 13, the planar-direction blurringcorrection section performing the planar-direction blurring correctionprocess by an electronic blurring correction process.
 16. Anon-transitory computer-readable medium storing a program that causes acomputer to function as: an image acquisition section that acquiresimages in time series; a coefficient calculation section that calculatesa correction coefficient for correcting blurring in a depth directionthat is a direction along an optical axis of an imaging section; and adepth-direction blurring correction section that performs adepth-direction blurring correction process on the images acquired intime series based on the correction coefficient, the depth-directionblurring correction process being a process that corrects blurring inthe depth direction; wherein the image acquisition section acquiring afirst image and a second image in time series, the second image beingsubsequent to the first image, the coefficient calculation sectioncalculating the correction coefficient that corresponds to amagnification of the second image with respect to the first image, andthe depth-direction blurring correction section performing thedepth-direction blurring correction process by correcting a change inmagnification of the images due to blurring in the depth direction basedon the correction coefficient.